SYSTEM AND METHOD FOR DEMOSAICING RAW DATA IMAGES WITH 
COMPRESSION CONSIDERATIONS, 

FIELD OF THE INVENTION 

The invention relates generally to the field of image processing, and more 
particularly to a system and method for demosaicing raw data (mosaiced) images. 

BACKGROUND OF THE INVENTION 

Color digital cameras are becoming ubiquitous in the consumer 
marketplace, partly due to progressive price reductions. Color digital cameras 
typically employ a single optical sensor, either a Charge Coupled Device (CCD) 
sensor or a Complementary Metal Oxide Semiconductor (CMOS) sensor, to 
digitally capture a scene of interest. Both CCD and CMOS sensors are only 
sensitive to illumination. Consequently, these sensors cannot discriminate 
between different colors. In order to achieve color discrimination, a color filtering 
technique is applied to separate light in terms of base colors, typically red, green 
and blue. 

A common filtering technique utilizes a color-filter array (CFA), which is 
overlaid on a sensor array, to separate colors of impinging light in a Bayer pattern. 
The Bayer pattern is a periodic pattern with a period of two different color pixels 
in each dimension (vertical and horizontal). In the horizontal direction, a single 
period includes either a green pixel and a red pixel, or a blue pixel and a green 
pixel. In the vertical direction, a single period includes either a green pixel and a 
blue pixel, or a red pixel and a green pixel. Therefore, the number of green pixels 
is twice the number of red or blue pixels. The reason for the disparity in the 
number of green pixels is because the human eye is not equally sensitive to all 
three primary colors. Consequently, more green pixels are needed to create a color 
image of a scene that will be perceived as a 'true color" image. 
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Due to the CFA, the image captured by the sensor is therefore a mosaiced 
image, also called "raw data" image, where each pixel only holds the value for 
either red, green or blue. The raw data image can then be demosaiced to create a 
color image by estimating the actual color value, the combination of red, green 
and blue, for each pixel of the image. The color value of a pixel is estimated by 
using color information from surrounding pixels. 

There are a number of conventional demosaicing methods to convert a raw 
data image into a color image. Three main common categories of demosaicing 
methods include interpolation-based methods, feature-based methods, and 
Bayesian methods. The interpolation-based demosaicing methods use simple 
interpolation formulas to interpolate the color planes separately. The 
interpolation-based demosaicing methods include bi-linear methods, band-limited 
interpolation methods using sincQ functions, spline interpolation methods, and the 
like. The feature-based demosaicing methods examine local features of a given 
image at the pixel level, and then interpolate the image accordingly. The basic 
idea of the feature-based methods is to avoid interpolating across edges of 
features. The Bayesian methods attempt to find the most probable color image, 
given the data, by assuming some prior knowledge of the image structure. 

After the raw data image has been demosaiced, the image is usually 
processed through a color-conversion operation and tone mapping, which are part 
of the image pipe-line. The resulting image is then typically stored in the camera 
using some sort of image compression, such as JPEG or JPEG-like compression 
schemes, to reduce the size of the image file. Therefore, the digital image that is 
eventually downloaded from the digital camera by the user is usually a compressed 
image file. 

Since the compression process is performed subsequent to the demosaicing 
process, some image enhancements achieved as a result of the demosaicing 
process may be significantly reduced or completely off-set by the compression 
process. As an example, the demosaicing process may create/predict high 
frequency components in the signal to produce a sharper image. However, the 
compression process may eliminate or reduce high frequency components of the 
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input image due to the use of quantizers. Therefore, any advantage gained in the 
demosaicing process may be negated by the compression process. 

In view of the above concern, there is a need for a system and method for 
efficiently processing digitally captured images such that the demosaicing process 
complements the subsequent compression process. 

SUMMARY OF THE INVENTION 

A system and method for processing mosaiced images utilizes a 
compression-aware demosaicing process that takes into consideration a 
subsequent compression process. The compression-aware demosaicing process is 
performed using a compression-considered demosaicing operator that incorporates 
a color space conversion operator and a frequency-based transformation operator, 
which are typically associated with the compression process. Consequently, the 
overall efficiency of the system and method is significantly increased. 
Furthermore, the compression-aware demosaicing process produces artifacts that 
complement the artifacts produced by the subsequent compression process such 
that the artifacts are less visible in the final color images, which increases the 
quality of the final color images. 

A method of processing a mosaiced image in accordance with the present 
invention includes steps of receiving the mosaiced image, which is a 
representation of a scene of interest, and processing the mosaiced image using a 
demosaicing operator on blocks of the mosaiced image to derive a representation 
of a demosaiced image. The demosaicing operator incorporates a frequency-based 
transformation operator to take into account a subsequent frequency-based 
compression process. The demosaicing operator may also incorporate a color 
space conversion operator. The color space conversion operator may be an 
operator for converting from the RGB color space to the luminance, red 
chrominance and blue chrominance (Yc,c b ) color space. The method may also 
include a step of compressing the representation of the demosaiced image using a 
frequency-based compression scheme, such as a DCT-based compression scheme 
or a wavelet-based compression scheme. 
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The frequency-based transformation operator incorporated in the 
demosaicing operator may be a DCT-based transformation operator. In another 
embodiment, the frequency-based transformation operator is a wavelet-based 
transformation operator. The frequency-based transformation operator, as well as 
the demosaicing operator, may be matrices. 

In one embodiment, the step of processing the mosaiced image is a step of 
interpolating the mosaiced image using the demosaicing operator, which is derived 
by defining selected coefficients of transformation-related coefficients as being 
equal to zero. The selected coefficients may be higher frequency components than 
remaining coefficients of the transformation-related coefficients. In another 
embodiment, the step of processing the mosaiced image is a step of interpolating 
the mosaiced image using the demosaicing operator, which is derived by defining 
the transformation-related coefficients as having a predefined probability 
distribution. The predefined probability distribution may be a Normal 
distribution. In this embodiment, the demosaicing operator may be derived using 
the Bayesian rule. 

The representation of the demosaiced image derived from the step of 
processing the mosaiced image may include a number of image pixel values of the 
demosaiced image. In another embodiment, the representation of the demosaiced 
image includes a number of transformed coefficients of the demosaiced image. 

A system for processing a mosaiced image in accordance with the present 
invention includes a compression-aware demosaicing module and a compression 
module. The compression-aware demosaicing module is configured to demosaic 
the mosaiced image to produce a representation of a demosaiced image using a 
demosaicing operator that incorporates a frequency-based transformation operator. 
The compression module is configured to compress the representation of the 
demosaiced image to produce a compressed image file by performing a frequency- 
based compression process. In an embodiment, the compression-aware 
demosaicing module and the compression module are embodied in an application 
specific integrated circuit. 

In one embodiment, the demosaicing operator incorporates a color space 
conversion operator for converting to the Ycyc r color space. In another 
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embodiment, the frequency-based transformation operator incorporated in the 
demosaicing operator is a DCT-based transformation operator, and the frequency- 
based compression process performed by the compression means is a DCT-based 
compression process. In another embodiment, the frequency-based transformation 
operator is a wavelet-based transformation operator, and the frequency-based 
compression process is a wavelet-based compression process. The frequency- 
based transformation operator, as well as the demosaicing operator, may be 
matrices. 

The demosaicing operator may be derived by defining transformation- 
related coefficients that are associated with the frequency-based compression 
process performed by the compression module. In one embodiment, the 
demosaicing operator is derived by defining selected coefficients of the 
transformation-related coefficients as being equal to zero. In another embodiment, 
the demosaicing operator is derived by defining the transformation-related 
coefficients as having a predefined probability distribution, which may be a 
Normal distribution. In this embodiment, the demosaicing operator may be 
derived using the Bayesian rule. 

The representation of the demosaiced image produced by the demosaicing 
means may include a number of image pixel values of the demosaiced image. In 
another embodiment, the representation of the demosaiced image includes a 
number of transformed coefficients of the demosaiced image. 

Other aspects and advantages of the present invention will become 
apparent from the following detailed description, taken in conjunction with the 
accompanying drawings, illustrated by way of example of the principles of the 
invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of an image processing system that performs 
compression-aware demosaicing process in accordance with the present invention. 

Fig. 2 is a block diagram of a compression-aware demosaic module 
included in the image processing system of Fig. 1. 
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Fig. 3 is a block diagram of a compression module included in the image 
processing system of Fig. 1. 

Fig. 4 illustrates various norms that can be used to generate the 
compression-considered demosaicing matrix for a DCT band-limited interpolation 
method in accordance with a first embodiment of the invention. 

Fig. 5 is a block diagram of a matrix generating system that can be used to 
generate the compression-considered demosaicing matrix in accordance with the 
first embodiment of the invention. 

Fig. 6 is a block diagram of a matrix generating system that can be used to 
generate the compression-considered demosaicing matrix in accordance with a 
second embodiment of the invention. 

Fig. 7 is a block diagram of an image processing system in accordance 
with an alternative embodiment of the invention. 

Fig. 8 is a process flow diagram of a method of processing a mosaiced 
image in accordance with the present invention. 

DETAILED DESCRIPTION 

With reference to Fig. 1, an image processing system 100 in accordance 
with the present invention is shown. The image processing system operates to 
digitally capture a scene of interest as a mosaiced or raw data image. The 
mosaiced image is then demosaiced and subsequently compressed for storage by 
the system. The image processing system utilizes a compression-aware 
demosaicing procedure that takes into consideration the fact that the demosaiced 
image is subsequently compressed. As a result, the compression-aware 
demosaicing procedure produces artifacts that complement the artifacts produced 
by the subsequent compression procedure. Thus, the overall visual distortion 
caused by the combined artifacts is significantly reduced, which increases the 
quality of the final digital image. In addition, the compression-aware demosaicing 
procedure performs steps that are typically performed during a compression 
process, which increases the efficiency of the image processing system to process 
the captured image. 
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The image processing system 100 includes an image capturing unit 102, an 
image pipeline unit 104, and a storage unit 106. The image capturing unit 
includes a sensor and a color-filter array (CFA). The sensor may be a Charge 
Coupled Device (CCD) sensor, a Complementary Metal Oxide Semiconductor 
(CMOS) sensor, or other type of photo-sensitive sensor. In an exemplary 
embodiment, the CFA includes red, blue and green filters arranged in a Bayer 
filter pattern. However, the CFA may include filters of other colors arranged in a 
different filter pattern. The image capturing unit operates to digitally capture a 
scene of interest as a mosaiced or raw data image having a color pattern in 
accordance with the filter pattern of the CFA, e.g., the Bayer pattern. 

The image pipeline unit 104 of the image processing system 100 includes a 
compression-aware demosaic module 108, a color and tone module 1 10, and a 
compression module 1 12. These modules of the image pipeline unit represent 
functional blocks and are not necessarily separate components. Preferably, the 
image pipeline unit is an application specific integrated circuit (ASIC). 

The compression-aware demosaic module 102 of the image pipeline unit 
1 04 operates to demosaic a raw data image into a demosaiced or color image. The 
demosaicing process is performed such that the subsequent compression process 
performed by the compression module 1 12 is taken into consideration. The 
components of the compression-aware demosaicing module are shown in Fig. 2. 
The compression-aware demosaicing module includes an image partitioning unit 
202, an interpolation unit 204, and a compression-considered demosaicing matrix 
206. The image partitioning unit operates to partition an input mosaiced image 
into image blocks, which are equivalent to the blocks that will be operated on by 
the compression module. As an example, the image partitioning unit may 
partition the input mosaiced image into 16x16 pixel image blocks. The 
interpolation unit operates to demosaic each image block of the mosaiced image 
by multiplying the image block with the compression-considered demosaicing 
matrix 206 to produce a demosaiced image. Thus, the compression-considered 
demosaicing matrix 206 is a demosaicing operator that interpolates a mosaiced 
image into a demosaiced image. 
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In a first embodiment, the compression-considered demosaicing matrix 
206 is derived using a Discrete Cosine Transform (DCT) band-limited approach. 
Thus, in this embodiment, the interpolation unit can be said to perform a 
demosaicing process using a DCT band-limited interpolation method. In a second 
embodiment, the compression-considered demosaicing matrix is derived using a 
DCT Bayesian approach. In this embodiment, the interpolation unit can be said to 
perform a demosaicing process using a DCT Bayesian method. As described in 
detail below, the compression-considered demosaicing matrix incorporates a DCT 
transformation operator, a sampling operator and a color conversion matrix. The 
operations associated with these operators are conventionally performed as part of 
the compression process. Since the compression-considered demosaicing matrix 
incorporates such operators, some or all of the operations associated with these 
operators need not be performed during the subsequent compression process. 

The compression-considered demosaicing matrix 206 may be stored in a 
read-only memory (not shown) associated with the image pipeline unit 104. 
Alternatively, the compression-considered demosaicing matrix may be stored in 
the storage unit 1 06. 

The color and tone module 1 10 of the image pipeline unit operates to 
perform color and tone enhancements to a demosaiced image from the 
compression-aware demosaic module 108. These enhancements are conventional 
operations and thus, are not described herein. The color and tone module may 
perform other conventional image operations such as flare-correction and 
intermediate color conversion. The color and tone module is optional and may be 
deleted from the image pipeline unit 104. 

The compression module 1 12 of the image pipeline unit 104 operates to 
compress the demosaiced image to a compressed image file using a DCT-based 
compression scheme, such as the JPEG compression scheme. The compression 
module is described herein as using the JPEG compression scheme. For reference, 
a conventional JPEG compression scheme is briefly described. 

For an input RGB image (e.g., a demosaiced image), each pixel of the RGB 
image is transformed into the luminance, red chrominance and blue chrominance 
(YCfCt,) color space. The space components of the Yc b c r color space are then 
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sampled, for example, at 4: 1 : 1 . The Y space component is not altered. However, 
the Cb and c r space components are sampled by a factor of two in each axis. Thus, 
the number of ct and c r samples is reduced by a factor of four. This is in line with 
the human visual perception which is less sensitive to sharp variations in color 
information. 

Next, each of the Yctc r color planes is partitioned into disjoint blocks of 
8x8. Each of these blocks is then processed in the following manner. The 8x8 
block of scalars is transformed using the DCT into a set of 64 coefficients, which 
can be arranged as an 8x8 array. The set of 64 DCT coefficients is then quantized 
using a uniform quantizer, which utilizes different quantization step sizes. In 
general, the higher frequency coefficients are quantized using larger quantization 
step size. Quantization is a non-linear, non-invertible operation. The quantized 
coefficients are then coded using a known coding technique, such as Huffman 
tables or arithmetic coding. Lastly, the coded data is wrapped in a standard file 
format. 

An important feature of the JPEG compression procedure is that the 
compression scheme assumes that the DCT coefficients that are quantized to a 
value of greater than zero are concentrated in the low frequencies. Another 
important feature is that the Y component space has twice the bandwidth of the c b 
and c r component spaces. Furthermore, all the operations of the procedure are 
performed on blocks of the image. The operations are performed on 16x16 blocks 
of the original image, and 8x8 blocks in the transformed/sub-sampled domain. 
Note that the partitioning of the image into blocks can be done before the color- 
conversion and sub-sampling, as these steps are point-wise operations that do not 
interact across block boundaries. 

Turning to Fig. 3, the compression module 1 12 of the image pipeline unit 
104 includes an image partitioning unit 302, a DCT transformation unit 304, a 
quantization unit 306, and a coefficient coding unit 308. The image partitioning 
unit performs the image partitioning operation of the JPEG compression scheme. 
As an example, the image is partitioned into 8x8 blocks of scalars. The DCT 
transformation unit performs the DCT transformation of the compression scheme. 
The quantization unit performs the quantization operation of the compression 
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scheme. Lastly, the coefficient coding unit performs the coding operation of the 
compression scheme. Thus, the compression module does not perform the color 
conversion and sub-sampling operations of the JPEG compression scheme, which 
have been included in the demosaicing process performed by the compression- 
aware demosaic module 108. 

The compressed image file produced by the compression module 1 12 is 
then stored in the storage unit 106 of the image processing system 100. The 
storage unit may be a conventional storage memory, such as DRAM or flash 
memory. Alternatively, the storage unit may be a drive that interfaces with a 
removable storage medium, such as a standard computer floppy disk. 

The image capturing unit 102, the image pipeline unit 104, and the storage 
unit 106 of the system 100 may be included in a single device, such as a digital 
camera. Alternatively, the image capturing unit may be included in a separate 
device. In this alternative embodiment, the functions of the image pipeline unit 
104 and the storage unit 106 may be performed by a computer. 

The derivation of the compression-considered demosaicing matrix 206 is 
now described. In accordance with the first embodiment, the compression-aware 
demosaic module 108 of the image pipeline unit 104 performs a DCT band- 
limited interpolation using the compression-considered demosaicing matrix to 
demosaic raw data images. For this embodiment, the compression-considered 
demosaicing matrix is derived by taking into consideration the band-limiting 
effect of the subsequent DCT-based compression process performed by the 
compression module 112. 

The DCT band-limited interpolation method essentially solves the problem 
of how to effectively interpolate a signal from its samples, when the signal is 
known to be band-limited in the DCT domain. The problem and its solution are 
described herein mathematically. To make the explanation more lucid, the 
description deals with one-dimensional signals. Extensions to the two- 
dimensional case with three color components will be described further below. 

Let x e 9*" be a signal described as a column vector: 

3c = *(«), A7 = 1,...,AT. (1) 
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This signal is sampled by taking every other element to get y e 9? 2 : 

y = {y(n)}={x(2n-l)l w = U.,y f (2) 
which can be written as 

y = [s}£, (3) 

N 

where [S] is an — x N sampling matrix. The matrix [S] is defined as 



[S]=S(iJ) = < 



N 

1 Forj = 2i-l, / = l,...,y (4) 
0 Otherwise 

Note that an extension to non-uniform sampling is straight forward. 

Let the NxN DCT transformation matrix [7], where each row is a basis 
function, be defined as: 

I -j= 9 Jfc = 0, 0<l<N-\ 

[T]=T(k,l) = \ (5) 

cos— i — , \<k<N-l 0<l <N-\ 

2N 



The transformation matrix [7] is a frequency-based transformation operator. Note 
that [7] is unitary and real. Namely, 

[TfT] =[T][T]=[I\ (6) 

where [I] is the identity matrix and the superscript "t" denotes the transpose of a 
matrix. Thus, the forward and backward DCT of the vector x can be written as 
Forward transform (Analysis): x c = [rjx, (7) 

Backward transform (Synthesis): x c = \t]x c . (8) 
Note that x c9 the DCT coefficients vector, is the same length (AO as the original 
signal. 

Let a signal q be defined to be band-limited and denoted as 

q e B k0 iff far(Jfc) - 0 for k > k 0 } (9) 

Using this notation, x will now be defined to be DCT band-limited, and denoted 

as 

xeD k0 iff {x c =[r]ic, x c eBj. (10) 
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The band-limitedness of the DCT coefficients is assumed on the DCT of 
the image when in YcbC r color space. Thus, a 3x3 transformation matrix from 
RGB values to Yct,c r values is needed. Assuming the RGB space is well defined, 
the transformation matrix can be readily defined. In general, the transformation 
matrix will depend on the RGB domain of the image. Where RGB values are 
assumed to be in the range [0,1], a sample transformation is: 



Y~ 




" 65.481 


128.553 


24.966 " 


~R~ 




"16" 


c b 




-37,797 


- 74.203 


112 


G 


+ 


128 






112 


-93.786 


-18.214 


B 




128 



The offset of the transformation is ignored, and the color transformation matrix 
from RGB to YcbCr will be denoted as [C r 2y\- The inverse transformation will be 
denoted as [C^r]. These color transformation matrices are color space conversion 
operators that can transform signals between an RGB color space and a YcbC r color 
space. 

The reconstruction problem to be solved can now be defined as follows: 
Given y , find x such that (\C rly ]x) e , and y = [S]x . (12) 

Note that since the original signal, x , is a discrete signal, the problem is much 
simpler than in the continuous case. However, this formulation is not yet simple 
enough, since the problem deals with a constrained solution. Namely, the desired 
formulation is one that has a solution of a linear set of equations, where the 
solution is constrained to a subspace. 

The problem defined in expression (12) can be written in the following 
equivalence: 

([C r2y ]x) gD^ iff x = [C y2r ][T]'a, and aeB^. (13) 
This enables the constrained problem defined in expression (12) to be replaced 
with the following unconstrained problem: 

Given y , find a € B^ such that y = [S][C y2r ][T] 1 a . (14) 
The compression-considered demosaicing matrix 206 is thus given by: 

[[S][C y2r ][T]'Y\ (15) 
which is the inverse of the matrix [S][C y2r ][T]' from the equation in the problem 
(14). As used herein, the expression "inverse of a matrix" is either a direct inverse 
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or pseudo-inverse of the matrix. Thus, the expression "inverting a matrix" refers 
to deriving the inverse of the matrix by either direct inverse or pseudo-inverse. 

The unconstrained problem (14) has substituted the problem of finding 
x g with the problem of finding a set of coefficients. Depending on the 

values of ko relative to the number of samples, which is N/2 in the current 
example, the above problem is either under-determined (ko>N/2), over-determined 
(ko<N/2\ or has the same number of unknowns and equations (ko=N/2). 
According to this classification, the problem can be solved using minimum-norm 
solution, least-squares solution, or exact inversion. 

For two-dimensional signals, there are a number of implementation issues. 
First, in order to enable a simpler linear-algebra formulation, the column-ordering 
of the image can be considered. The column order of a matrix can be obtained by 
concatenating the columns of the matrix to form a vector. Thus, each color-plane 
can be ordered as one column vector. Second, since the DCT is a separable 
transform, the DCT can be applied first to rows and then to columns, or vice versa. 
Moreover, using the Kronecker product, the transformation matrix on a column- 
order version of the matrix can easily be defined. The last implementation issue 
involves the frequency of the input signal. A one-dimensional signal has only 
one-linear axis of frequency and the frequency is a scalar. Thus, the issue of 
frequency for a one-dimensional signal is straightforward. However, for two- 
dimensional signals, the issue of frequency is more complex, which is now 
described. 

Assuming a matrix [A] represents the DCT coefficients of a two- 
dimensional signal (e.g., one of the image color planes), the band-limitedness to k 0 
can be any of the following: 

I 1 , norm: A(kJ) = 0 for | * | + 1 / 1> k 0 . (16) 

L 2 norm: A(kJ) = 0 for ^k 2 + l 2 > k 0 . (17) 
L° norm : A(kJ) = 0 for max(*,/) > V < 18) 
The expressions (16), (17), and (18) are illustrated in Fig. 4, where DCT 
coefficients of a two-dimensional signal are presented as a two-dimensional set of 
scalars. The DCT coefficients are shown in Fig. 4 as cells in an 8x8 grid 402, 
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where each cell corresponds to one coefficient. The non-shaded cells correspond 
to the DCT coefficients that have been assumed to be zero. The remaining DCT 
coefficients are shown as shaded cells. As illustrated by Fig. 4, the band- 
limitedness depends on the norm used. The cells that are intersected by the 
boundary between the shaded region and the non-shaded region should be decided 
as either shaded or non-shaded cells. Alternatively, these intersected cells can be 
decided using strict inequality for the range. For the Y component, the L l norm is 
found to be more appropriate, due the manner in which the Y component is 
sampled and the fact that the G component of RGB color space corresponds best to 
the Y component of YcrCb color space. 

The compression-considered demosaicing matrix 206 is generated by a 
separate system and then programmed into the image processing system 100. In 
Fig. 5, a matrix generating system 500 in accordance with the first embodiment is 
shown. The matrix generation system is configured to generate the compression- 
considered demosaicing matrix, as defined by expression (15). The system of Fig. 
5 includes a matrix transposing module 502, a multiplication module 504 and a 
matrix inverting modules 506. Although the modules 502-506 may be 
implemented in any combination of hardware, firmware and software, these 
modules are preferably embodied in the matrix generating system as a software 
program that performs the functions of the modules 502-506 when executed by a 
processor (not shown). 

In operation, the matrix transposing module 502 of the matrix generation 
system 500 receives an input matrix [7], which is the DCT transformation matrix. 
The matrix transposing module then transposes the received matrix [7] to output a 
matrix [7]', which is transmitted to the multiplication module 504. The 
multiplication module 504 receives input matrices [5] and the [C r 2 y ] and the 
output matrix [7]' from the matrix transposing module 502. The input matrix [5] 
is the sampling matrix. The input matrix [C r 2 y ] is the color transformation matrix. 
The multiplication module then multiplies the three received matrices in the order 
illustrated in Fig. 5 to derive a matrix [S][C y2r ][T] , which is transmitted to the 
matrix inverting module 506. The module 506 then inverts the matrix 
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[SHC^llT] to generate the compression-considered demosaicing matrix 206, as 

defined by expression (15). 

In accordance with the second embodiment, the compression-aware 
demosaic module 108 of the image pipeline unit 104 performs a DCT Bayesian 
5 demosaicing process using the compression-considered demosaicing matrix 206 to 
demosaic raw data images. For this embodiment, the compression-considered 
demosaicing matrix is derived by assuming some a priori knowledge about the 
probability distribution of the DCT coefficients. 

The DCT Bayesian approach method associates a probability distribution 
10 function with each DCT coefficient. Furthermore, the DCT Bayesian approach 
method assumes that the coefficients are jointly Normal, with mean vector J} c and 
covariance matrix [Rc] 9 or expressed mathematically: 

K =([T][C r2y )x) ~ N(fi e9 [R 9 ]). (19) 
The Bayesian approach is based on the Bayesian rule for probabilistic 
15 distributions. The Bayesian rule is expressed as: 

p^^Pr^Pr^ (2Q) 

My) 



The following notations for general Normal distributions are used herein: 
□ Mean: Jl c =E{z c ) (21) 

Covariance: [R e ]=E(z e - ft c )(z c - //,)' (22) 



20 Normal distribution: Pr(f f ) = Pr(z c (l),z c (2),...,z c (A0) (23) 

1 



V(2*)"IMJI 

(24) 

The DCT Bayesian approach uses an additive noise model to formulate the 
reconstruction problem. The additive noise model is expressed as: 
25 y = [S]x + it, (25) 

where n has a Normal distribution with covariance matrix [R„], 

n~N(0,[R n ]). (26) 
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The above additive noise model is appropriate since it is natural to assume the 
measurements are noisy. 

Using the above assumptions, the problem of reconstruction can now be 
formulated as follows: 

5 Given y, such that y = [S]x + «, where n ~ N(0, [R n ]), (27) 

find Jc, assuming z c = ([T][C r2y ]x) ~ N(£ C ,[R C ]). 

The problem (26) is solved using the Bayesian Rule (19). Since the sampled data 
y is assumed to be given, x is chosen such that: 

x = arg max{Pr(j>/jc) Pr(jc)} . (28) 

X 

10 The assumption about the Gaussian model for noise is incorporated with the 
Gaussian model for the coefficients. With respect to noise, according to 
expression (27), 

Pr(y/x) = Pr(n = y-[S]x). (29) 
Using expressions (23) and (25), equation (28) can be written as: 
1 5 log(Pr(^ /*)) = c-(y- [S]xY [R n V(y- [S]x) , (30) 

where c is a constant, independent of x . The X A factor is dropped, since this factor 
is eventually cancelled. With respect to the coefficients, the assumption is that 
the coefficients in the transformed domain have a known Normal distribution. In 
addition, a linear transformation of a Normal vector is Normal. Thus, the 
20 probability of the original discrete signal x can be written as: 

Pr(x) = Pr([C y2r ][Tyz c ) (31) 

~*([C, 2 j[r]';u^ (32) 

= (33) 
The above probability can then be written in the following log form: 

25 log(Pr(x)) = c - (x - fi x )' [R x ]" 1 (* - M x ) - (34) 

Inserting equations (29) and (33) into equation (27), the following equation is 
derived: 

log(Pr(J?/x)Pr(x)) (35) 

= c - (y - [S]x)' [R n ]" 1 (y - [S]x) - (5 - fi x )' [R J" 1 (5 - fi x ) . (36) 
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The covariance matrices are positive and symmetric. Therefore, the same holds 
true for [R n ]~ l , [R x ]~ l , and [5]' [/?„]'' [5] . Consequently, expression (34) can be 
written as: 

= c' + 2x' [ST [/?„]-' y + 2x' [R x ]"' fi, 

5 "*'([* J"' [*„]-'[$])* (37) 

= c' + 2x'([Sy[R n ]- , y + [K i r , fi,) 

-*'([*,]-' +[S]'[* < J- 1 [S])5 ' (38) 

t c ' + x'p-x'[Q]x, (39) 

where 

10 P = 2([S]' [R„ T l y + [R x ]~ x Mx) and (40) 

[Q]=(.[R x r l +[sy[R n r l [s]). m 

Below are two derivative rules for vectors: 

■4- (x ' [A]x) = [A]x + [A]'x and (42) 

cbc 

-^(x'a) = a. (43) 
cbc 

15 Applying these rules (41) and (42) to expression (38), the following equation is 
derived: 

|r log(Pr(y/x) Pr(x)) = p - [g]5 - [g]' x . (44) 

Equating to zero and solving for x , 

3c = ([G] + [G] , )" l P- («) 
20 Note that this is the maximum point because [Q] is positive and therefore, the 
Hessian is negative. Since [Q\-[Q\\ equation (44) can be written as: 

x = \[QVp- (46) 

The above equation can be expanded to yield the following simpler form for 
computation: 

25 x = [QV [Sy [R n Y l y + [Q]~ l [R x ]~ l fi x . (47) 

The compression-considered demosaicing matrix 206 is thus given by: 
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The term [£>]"* [/?J 1 /J x of equation (46) represents bias. Assuming known noise 
and known distribution of the coefficients, equation (46) gives a closed form 
solution to find the most probable x , given the measurements y . 

The level of noise can be assumed as needed. If the noise is assumed to be 
very low, the resulting image at the sampling points is essentially forced to be 
exactly equal to the sample values. However, if the noise is assumed to be high, 
the values at the sampling points are regarded merely as recommended values to 
the algorithm. Note that noise can be estimated for the specific type and 
arrangement of components included in the system. The distribution of the 
coefficients may be estimated using DCT models or by computing statistics from a 
number of sample images. 

The bias term [Q]~ l [R x ]~ l fi x in equation (48) can be determined off-line. 
In practice, this bias term can be added after image blocks have been multiplied by 
the compression-considered demosaicing matrix 206 by the interpolation unit 204 
of the compression-aware demosaic module 108. 

In Fig. 6, a matrix generating system 600 in accordance with the second 
embodiment is shown. The matrix generation system is configured to generate the 
compression-considered demosaicing matrix 206, as defined by expression (48). 
The system of Fig. 6 includes matrix inverting modules 602, 604 and 606, a 
matrix transposing module 608, multiplication modules 610 and 612, and a 
summing module 614. Although the modules 602-614 may be implemented in 
any combination of hardware, firmware and software, these modules are 
preferably embodied in the matrix generating system as a software program that 
performs the functions of the modules 602-614 when executed by a processor (not 
shown). 

In operation, the matrix inverting module 602 of the matrix generation 
system 600 receives an input matrix [R x ]. The input matrix [R x ] is the covariance 
matrix of x (original discrete signals), which can be estimated by using various 
conventional methods, such as by using DCT models or by computing statistics 
from a number of sample images. The module 602 then inverts the received 
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matrix [R x ] to output a matrix [R x ]'\ which is transmitted to the summing module 
614. The matrix inverting module 604 receives an input matrix [R„]. The input 
matrix [R„] is the covariance matrix of n (noise), which can be estimated for the 
specific type and arrangement of components included in the image processing 
system 100 of Fig. 1. The module 604 then inverts the received matrix [R n ] to 
output a matrix [R n ]'\ which is transmitted to the multiplication modules 610 and 
612. The matrix transposing module 608 receives an input matrix [5]. The matrix 
transposing module then transposes the received matrix [S] to output a matrix [S]*, 
which is also transmitted to the multiplication modules 510 and 512. 

The multiplication module 610 receives the two output matrices [S]' and 
[R„]' } from the matrix transposing module 608 and the matrix inverting module 
604, respectively, and the input sampling matrix [S]. The multiplication module 
610 then multiplies the three received matrices in the order illustrated in Fig. 6 to 
derive a matrix [S]'[/?„] which is transmitted to the summing module 614. 
The summing module receives the matrix [S]'[/? w ] ~ ] [S] from the multiplication 
module 610 and the matrix [R x ]~ l from the matrix inverting module 602. The 
summing module then adds the two received matrices to output a matrix 
[Rx]' } +[S}'[Rn] which is transmitted to the matrix inverting module 606. The 
matrix [/?jc]" i+ [5] / [^«][5] is equivalent to the matrix [Q] 9 as defined in expression 
(41). Therefore, the matrix [/?J _I +[5]'[^rt][5] will now be referred as the matrix 
[Q\- The module 606 inverts the received matrix [Q] and output a matrix [Q]'\ 
which is transmitted to the multiplication module 612. The module 612 receives 
the matrix [Q\' x from the matrix inverting module 606, the matrix [S]' from the 
matrix transposing module 608, and the matrix [R n ]' } from the matrix inverting 
module 604. The multiplication module 612 then multiplies the three received 
matrices in the order illustrated in Fig. 6 to generate the compression-considered 
demosaicing matrix 206, as defined by expression (48). 

With reference to Fig. 7, an image processing system 700 in accordance 
with an alternative embodiment is shown. The image processing system 700 
includes a number of components found in the image processing system 100, 
which are shown in Figs. 1, 2 and 3. These common components are indicated in 
Fig. 7 by the same reference numerals. The image processing system 700 of Fig. 7 
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includes the image capturing unit 102, an image pipeline unit 702 and the storage 
unit 106. The image pipeline unit 702 includes a compression-aware demosaic 
module 704 and a compression module 706. Similar to the compression-aware 
demosaic module 108 of the system 100 of Fig. 1, the compression-aware 
demosaic module 704 includes the image partitioning unit 202 and the 
interpolation unit 204. However, the compression-aware demosaic module 704 
utilizes a different compression-considered demosaicing matrix 708. Using the 
compression-considered demosaicing matrix 708, the compression-aware 
demosaic module 706 outputs a set of DCT coefficients for each image block 
processed by the interpolation unit 204. In effect, the interpolation unit 204 
performs DCT transformation, as well as demosaicing. Thus, the compression 
module 706 needs only to perform quantization and coding to complete the 
compression process. Consequently, the compression module includes only the 
quantization unit 306 and the coefficient coding unit 308. 

The compression-aware demosaic module 704 can perform either the DCT 
band-limited interpolation method or the DCT Bayesian approach method, 
depending on the compression-considered demosaicing matrix 708. For the DCT 
band-limited interpolation method, the compression-considered demosaicing 
matrix 708 is derived by multiplying the compression-considered demosaicing 
matrix 206, as defined in expression (15), with the DCT transformation matrix 
[7]. The compression-considered demosaicing matrix 708 is thus given by: 

[T)[[S][C y2r )[T]'r. (49) 
For the DCT Bayesian approach method, the compression-considered 
demosaicing matrix 708 is derived by multiplying the compression-considered 
demosaicing matrix 206, as defined in expression (48), with the DCT 
transformation matrix [7]. The compression-considered demosaicing matrix 708 
is thus given by: 

[T][Qr l [S]'[R n Y l - < 5 °) 
The compression-considered demosaicing matrix 708 in accordance with 
the DCT band-limited interpolation method, as defined by expression (49), can be 
generated by a modified version of the matrix generation system 500 of Fig. 5. 
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The modified matrix generation system includes a second multiplication module 
(not shown) connected to the output of the matrix inverting module 506. Thus, 
the second multiplication module receives the output from the matrix inverting 
module 506. In addition, the second multiplication module receives an input 
matrix [7], which is multiplied to the output of the matrix inverting module 506. 
The output of the second multiplication module is the compression-considered 
demosaicing matrix 708, as defined by expression (49). 

The compression-considered demosaicing matrix 708 in accordance with 
the DCT Bayesian approach method, as defined by expression (50), can be 
generated by a modified version of the matrix generation system 600 of Fig. 6. 
The modified matrix generation system is configured to receive an input matrix 
[7] at the multiplication module 612, in addition to the outputs from the matrix 
inverting modules 604 and 606, and from the matrix transposing module 608. The 
multiplication module 612 thus multiplies the matrices [7], [Q]'\ [R n ]'\ and [S]' 
to generate the compression-considered demosaicing matrix 708, as defined by 
expression (50). 

A method of processing a mosaiced image in accordance with the present 
invention is described with reference to Fig. 8. At step 802, a compression- 
considered demosaicing matrix is generated according to the DCT band-limited 
interpolation method or the DCT Bayesian approach method. At step 804, an 
input mosaiced image is received. Next, at step 806, the input mosaiced image is 
partitioned into image blocks. As an example, the input image may be partitioned 
into 16x16 pixel image blocks. At step 808, the input image is demosaiced using 
the compression-considered demosaicing matrix. Next, at step 810, the 
demosaiced image is color and tone enhanced. Step 8 1 0 is an optional step, and 
thus, may be omitted from the method. 

At step 812, the demosaiced image is partitioned into blocks for 
compression. The blocks are then DCT transformed, at step 814. As a result, 
DCT coefficients of the image are derived. Next, at step 816, the DCT 
coefficients are quantized. The quantized coefficients are then coded using, for 
example, Huffman tables or arithmetic coding, to derive a compressed image file, 
at step 818. Next, at step 820, the compressed image file is outputted. 
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In an alternative embodiment, steps 810, 812 and 814 are omitted from the 
method. In this embodiment, the DCT coefficients are generated by the 
demosaicing operation at step 808. Thus, in this alternative embodiment, steps 
808 and 812 are combined into a single step of demosaicing the input mosaiced 
image using the compression-considered demosaicing matrix to derive the DCT 
coefficients. 

The above-described demosaicing methods and the corresponding 
demosaicing matrices can be modified to be applicable to wavelet-based 
compression schemes, such as the JPEG 2000. For a wavelet-based compression 
scheme, the transformation matrix, as defined in equation (5), is replaced with a 
wavelet transformation matrix. The formulation and solution of the reconstruction 
problem will be the same. The resulting demosaicing matrix can then be used to 
demosaic input mosaiced images, prior to being compressed using the wavelet- 
based compression scheme. Consequently, the only significant modification 
needed to convert the image processing systems 100 and 700 to an image 
processing system that utilizes a wavelet-based compression scheme is the 
replacement of either the compression module 1 12 or the compression module 706 
with a comparable compression module that performs compression based on 
wavelets. 
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